Over the last decade, malicious software has become a pervasive problem for Internet users as many networked resources include vulnerabilities that are subject to attack. For instance, over the past few years, more and more vulnerabilities are being discovered in software that is loaded onto network devices, such as vulnerabilities within operating systems for example. While some vulnerabilities continue to be addressed through software patches, prior to the release of such software patches, network devices will continue to be targeted for attack by exploits, namely malicious computer code that attempts to take advantage of a vulnerability in computer software by acquiring sensitive information or adversely influencing or attacking normal operations of the network device or the entire enterprise network.
In particular, a malware writing technique known as ROP has become fairly widespread recently. ROP is an exploit that allows a writer of malware to chain together sequences of instructions through return instructions thereby accomplishing one or more tasks via the execution of the chain of sequences of instructions. ROP techniques were developed as a way to circumvent data execution prevention (DEP) techniques, which have been recently implemented in many operating systems to thwart unauthorized activities including malicious attacks.
A “DEP system” prevents the execution of portions of memory allocated by an application marked as “non-executable.” For instance, areas of allocated memory that contain data as opposed to executable code may be marked as “non-executable.” In particular, the stack and “virtual” heap of memory allocated by an application are typically marked as non-executable by default. Therefore, malware writers that previously inserted shellcode into the stack or virtual heap and executed an instruction to direct the execution flow to the inserted shellcode are not able to execute the inserted shellcode. A DEP system typically prevents malware writers from executing the inserted shellcode by causing the application to terminate.
In order to circumvent the protections established by a DEP system, malware writers turned to return-oriented programming Malware writers may accomplish tasks they would have inserted into the stack and/or virtual heap using shellcode by executing sequences of instructions already appearing in executable code, such as a dynamically-loaded library (DLL), loaded by the application. Using the ROP technique, malware writers search the areas of the allocated memory marked as “executable” (such as DLLs) for sequences of instructions that, chained together, accomplish any desired tasks. The sequences of instructions are chained together through the use of return instructions following the sequence of instructions. For example, the return instruction following sequence_1 will point to sequence_2. Therefore, merely performing a search of the stack or virtual heap for shellcode may not be sufficient to detect such exploits.